A Modular Visual Tracking System

I am currently building an active visual tracking system for a real world robot. The hardware is being built at MIT under the supervision of Professors Rod Brooks and Lynn Andrea Stein, and is humanoid in form. The software is also humanoid: I am basing its organization on models of early vision in the human brain. Most of the software is still in the design phase; what I describe here is the part of the system that is already up and running. The robot, named Cog, has roughly the same degrees of freedom in the waist, neck, arms and eyes as a human and is designed with similar proportions in mind. The eyes sport a simulated fovea each eye consists of two cameras mounted in the same plate, one with a wide field of view, and one with a much narrower view. Both cameras produce 128 x 128 gray scale images. The narrow one is used for tracking, and can be used for object recognition, while the wider view will be used for motion detection and peripheral vision. I have written a visual tracking system that will eventually be hooked up to the motors in Cog’s neck and eyes. For now, tracking is simulated by moving a small 16 x 16 “attention window” around the larger image from one of the cameras. On startup, the system memorizes the 16 x 16 segment in the center of the full image as a “reference” image. For the rest of the run, the system moves the window around to maintain whatever was initially present within its view. The tracking system runs as follows. Once a new frame is grabbed, the system makes a guess about where the new window location should be, based on its previous velocity. Next, a portion of the image slightly larger than the attention window is selected, and the derivative of this region is computed.’ Finally, a simple correlation is performed between the memorized reference image and the nine 16 x 16 windows centered around the pixel position nearest the guess. The one